This PR testing SPMD vLLM is in its early stages and not ready for immediate merging.
I am here just to confirm that the main-branch vLLM now works successfully with verl's test case (run_fsdp_vllm.py) and demonstrates compatibility. Below are some baseline comparisons for weight sync duration.

Configuration: 8*L20 GPUs, TP=4
A. Across Process(broadcast/gloo): run with python test_sync_weight_openrlhf.py with vllm_sync_backend = "gloo", rank 7 broadcast weights to rank0-3 with gloo backend.
B. Across Process(broadcast/nccl): run with python test_sync_weight_openrlhf.py with vllm_sync_backend = "nccl", rank 7 broadcast weights to rank0-3 with nccl backend.
C. FSDP+vLLM: the original test case, run 4 workers with torchrun --nproc-per-node=4 run_fsdp_vllm.py
D. FSDP+vLLM(spmd): using vllm='0.6.6.post2.dev252+g8027a724', run 4 workers with torchrun --nproc-per-node=4 run_fsdp_vllm_spmd.py

And the weight sync time (unit:second) is recorded as:

Note that the across-process weight sync only includes the broadcast component (the complete weight sync should also include weight gathering). FSDP+vLLM and FSDP+vLLM(spmd) should perform identically since the sync weight logic remains unchanged. Based on these results, I have two conclusions:

1. With SPMD vLLM now available, there's no need to explore cross-process weight synchronization methods.
2. I will test replacing the original vLLM with the new vLLM in the core logic to verify verl's compatibility.